Learning-based visual attention prediction system and method thereof

ABSTRACT

A learning-based visual attention prediction method is disclosed. The method includes a correlation relationship between the fixation density and at least one feature information being learned by training, followed by a test video sequence of test frames being received. Afterward, at least one tested feature map is generated for each test frame based on the feature information. Finally, the tested feature map is mapped into a saliency map, which indicates the fixation strength of the corresponding test frame, according to the correlation relationship.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to visual attention predictionsystems and methods, and more particularly to a learning-based visualattention prediction system and method for video signals.

2. Description of Related Art

Visual attention is an important characteristic of the five biologicalsenses. It helps the human brain filter out excessive visual informationand enables the eyes to focus on particular regions of interest. Visualattention has been a subject of research in neural science, physiology,psychology, and vision. Data gleaned from these studies can be used notonly to greatly enrich current understandings of the psychophysicalaspect of visual attention, but also to enhance the processing of thevideo signals.

The fixation points in an image usually attract the most attention. Ifthe attended regions of the image can be predicted, the video signals ofthe more attractive regions can be detail-processed and visually moreimportant areas can be better preserved in the coding process. A typicalvisual attention model consists of two parts: extraction of features andfusion of features. The feature maps are generated after featureextraction from the image, and the feature maps are then fused to form asaliency map by nonlinear fusion, linear fusion with equal weight, orlinear fusion with dynamic weight. However, improper weight assignmentin the feature fusion process or low-level features alone, such ascolor, orientation, etc., can result in perceptual mismatches betweenthe estimated salience and the actual human fixation.

For the reason that conventional 3D imaging systems could noteffectively predict visual attention, a need has arisen to propose anovel visual attention prediction system and method that can faithfullyand easily predict visual attention.

SUMMARY OF THE INVENTION

In view of the foregoing, it is an object of the embodiment of thepresent invention to provide a learning-based visual attentionprediction system and method to predict visual attention effectively.

According to one embodiment, a learning-based visual attentionprediction system which comprises a feature extraction unit and aregression model is disclosed. The feature extraction unit receives atest video sequence which comprises a plurality of test frames, andgenerates at least one tested feature map for each test frame based onat least one feature information. The regression model has a correlationrelationship between the fixation density and the feature information.The regression model maps the tested feature maps into a saliency map,which indicates the fixation strength of the corresponding test frame,according to the correlation relationship.

According to another embodiment, a learning-based visual attentionprediction method is disclosed. The method comprises the followingsteps: firstly, a correlation relationship between the fixation densityand at least one feature information is learned by training. Then, atest video sequence which comprises a plurality of test frames isreceived. Afterward, at least one tested feature map is generated foreach test frame based on the feature information. Finally, the testedfeature map is mapped into a saliency map, which indicates the fixationstrength of the corresponding test frame, according to the correlationrelationship.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram illustrating a learning-based visualattention prediction system according to one embodiment of the presentinvention;

FIG. 2 shows an architecture diagram illustrating an eye tracking systemaccording to one embodiment of the present invention;

FIG. 3 exemplifies the training video sequences according to oneembodiment of the present invention;

FIG. 4 exemplifies the training frames and the corresponding fixationmaps according to one embodiment of the present invention;

FIG. 5 exemplifies a training frame and the corresponding fixation mapand fixation density map according to one embodiment of the presentinvention; and

FIG. 6 shows a flow diagram illustrating a learning-based visualattention prediction method according to one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Referring more particularly to the drawings, FIG. 1 shows a blockdiagram illustrating a learning-based visual attention prediction systemaccording to one embodiment of the present invention. The visualattention prediction system 1 comprises a fixation data collection unit11, a feature extraction unit 13, a fixation density generator 15, atraining sample selection unit 17, a training unit 18, and a regressionmodel 19. The computational scheme of the visual attention predictionsystem 1 is executed in a training phase and a test phase.

During the training phase, it must obtain the training samples and thefixation data from the given training video sequences, and then learnmapping information needed in the test phase. Firstly, the fixation datacollection unit 11 receives a plurality of training video sequences 3a-3 h, as shown in FIG. 3, and each training video sequence comprises aplurality of training frames. The fixation data collection unit 11detects a plurality of fixation points gazed in each training frame ofthe training video sequences 3 a-3 h, so as to collect all fixationpoints of each training frame to generate a fixation map.

Specifically, the fixation data collection unit 11 comprises an eyetracking system, as shown in FIG. 2. The eye tracking system comprises ahost PC 111, a displayer 113, a supporting rest (or a chin and foreheadrest) 115 and a camera 117. The displayer 113 is configured to displaythe training video sequences 3 a-3 h. The supporting rest 115 isconfigured to support the viewer 2 to watch the training video sequences3 a-3 h displayed by the displayer 113. The camera 117 is configured tofront the viewer 2 and tracks the eye movement of the viewer 2. The hostPC 111, coupled with the displayer 113 and the camera 117, controls thedisplayer 113 to display the training video sequences 3 a-3 h andrecords the fixation points (human fixation data) which the viewer 2gazes in the training frame. In practice, plural viewers 2 withdifferent backgrounds were invited to participate in the eye trackingexperiment, and the training video sequences 3 a-3 h would be displayedsequentially by the displayer 113. As the viewer's chin and head werewell seated on the supporting rest 115, the camera 117 started to detectthe position of the current displayed training frame where the viewer 2gazes (i.e. the fixation points) which are recorded by the host PC 111.Therefore, for each training frame, the fixation points of all viewersare collected to generate a fixation map.

With reference to FIG. 4, the training frames and the correspondingfixation maps are elucidated according to one embodiment of the presentinvention. The training video sequences 3 a, for example, consists ofplural successive training frames 3 a 1-3 a 5. The fixation points ofthe training frames 3 a 1-3 a 5 where the viewers 2 gaze are collectedto generate the corresponding fixation maps 4 a 1-4 a 5. For example,each fixation point in the fixation maps 4 a 1 represents the positionof the training frames 3 a 1 where the viewer 2 gazes. The empiricalfixation data collected by the eye tracking system from viewers 2 orhuman subjects are used as the ground truth for training the regressionmodel 19.

The fixation density generator 15, coupled to the fixation datacollection unit 11, is configured to transform each fixation map into afixation density map which represents the salience of each trainingframe. Specifically, the fixation map (e.g. 4 a 1) generated in the datacollection process for each training frame (e.g. 3 a 1) of any trainingvideo sequence (e.g. 3 a) is a set of discrete fixation points {(x_(n)^(f), y_(n) ^(f)), n=1, . . . , N}, wherein N represents the number ofthe viewers 2 participating in the eye tracking experiment. The fixationdensity generator 15 interpolates the fixation map to make a fixationdensity map. Please refer to FIG. 5, which exemplifies a training frameand the corresponding fixation map and fixation density map according toone embodiment of the present invention. As shown in FIG. 5, thefixation density generator 15 filters the fixation map 4 a 5 of thetraining frame 3 a 5 with a Gaussian distribution function, yieldingequation (1), so as to generate a fixation density map 5 b.

$\begin{matrix}{{{s\left( {x,y} \right)} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}{\frac{1}{2\pi \; \sigma_{s}^{2}} \cdot {\exp \left( {- \frac{\left( {x - x_{n}^{f}} \right)^{2} + \left( {y - y_{n}^{f}} \right)^{2}}{2\sigma_{s}^{2}}} \right)}}}}},} & (1)\end{matrix}$

Wherein, s(x, y) denotes the fixation density map 5 b which carries afixation density value of each pixel in the training frame 3 a 5. Andσ_(s), the standard deviation of the Gaussian distribution, isdetermined in accordance with the visual angle accuracy of the eyetracking system, σ_(s)=L×tan 0.5π/180, where L is the viewing distancebetween the viewer 2 and the displayer 113. We can see from equation (1)that the fixation density is estimated by taking the Gaussian weightedaverage of the fixation values. In this way, each fixation pixelpropagates its value to nearby pixels. Therefore, a pixel in a denselypopulated fixation area is more attractive than a pixel in a sparselypopulated fixation area. FIG. 5 a, which is the 3D view of the fixationdensity map 5 b, indicates the fixation strength distribution of eachpixel.

The feature extraction unit 13 receives the training video sequences 3a-3 h one by one and extracts the features of each training frame. Thefeature extraction takes into account what kind of feature in the imageattracts human attention. Based on three low-level feature information,such as color, motion, and orientation information of the video frameand one high-level feature information, such as face, the featureextraction unit 13 generates four corresponding training feature maps,such as the color feature map, motion feature map, orientation featuremap, and face feature map, for each training frame of the training videosequences 3 a-3 h.

The training feature map carries a training feature value of each pixelin the corresponding training frame. Specifically, according to colorcontrast, the color feature map denotes the degree of attraction of eachpixel (or block) in the image. The motion feature map denotes therelative motion of each pixel (or block) in the image. The larger themotion contrast, the stronger the response of the neural cells. Thecontrast of orientation, which is obtained by computing the differenceof the two local orientation images, forms the orientation feature map.And the human face in the image may be detected to form the face featuremap. The face detector may be implemented by a suitable conventionaltechnique, for example, as disclosed in “Face Detection using Local SMQTFeatures and Split Up SNoW Classifier” by Nilsson et al., the disclosureof which is hereby incorporated by reference.

In the embodiment of the present invention, for each training frame(e.g. 3 a 1) of any training video sequences (e.g. 3 a), the featureextraction unit 13 generates four feature maps and the fixation densitygenerator 15 generates one corresponding fixation density map. Afterobtaining the above information generated in training phase, thetraining unit 18 can train the regression model 19 according to thecorrelation relationship between the fixation density and the featureinformation of each pixel, so as to enter into a test phase later.However, if using all and huge amount of correlation relationshipsbetween the fixation density and the feature information of all pixelsto train the regression model 19, it is time consuming and inefficient.Therefore, before entering into the test phase, the training sampleselection unit 17 selects the training samples for regressor training.Specifically, each training sample is represented as a quintuplet ofdata consisting of a fixation density value and four correspondingfeature values of a pixel.

The training sample selection unit 17, coupled to the training unit 18,is configured to select at least one sample frame among the trainingframes of each training video sequence. The density of the fixationpoints of the selected sample frame should be most dense in a specificscene. Specifically, since the spatial fixation distribution of atraining frame reflects the degree of attention of the training frame,the training sample selection unit 17 finds the centroid of the fixationpoints for each training frame of a training video sequence andcalculates the mean of the distance between each fixation point and thecentroid. The frame with the smallest mean is selected as the sampleframe for representing the training video sequence in the specificscene. One may, but is not limited to, select more than one sample framefrom each training video sequence.

In another embodiment, the training sample selection unit 17 selects arelatively small number of pixels from each sample frame as the samplepixels. The selected sample pixels are the fixation points where inrelatively dense region of the fixation density map of the sample frame.Once the sample pixels are selected, the training unit 18 will train theregression model 19 according to mapping relationship between thefixation density values of the sample pixels and the training featurevalues of the sample pixels.

After the training samples are obtained, the regression model 19 istrained to learn the correlation relationship between the fixationdensity and the features information of the training samples. In oneembodiment, the training unit 18 adopts the support vector regression(SVR) algorithm to train the regression model 19. Besides learning thecorrelation relationship between the fixation density and the featuresinformation by receiving plural training video sequence real time, thecorrelation relationship can be pre-built in the regression model 19. Inpractice, the feature extraction unit 13 receives a test video sequencewhich comprises a plurality of test frames, and generates four testedfeature maps for each test frame based on four corresponding featureinformation. Then, the regression model 19 maps the tested feature mapsinto a saliency map, which indicates the fixation strength of thecorresponding test frame, according to the trained correlationrelationship. The saliency map, similar to the fixation density map 5 b,has a saliency region with relatively large fixation strength which isprediction of visual attention. Therefore, it needs to performrelatively detailed image process on the portion of test frame whichcorresponds to the saliency region.

FIG. 6 shows a flow diagram illustrating a learning-based visualattention prediction method according to one embodiment of the presentinvention. Firstly, plural viewers 2 were invited to participate in theeye tracking experiment to collect fixation data. In step S601, the hostPC 111 controls the displayer 113 to display the training videosequences 3 a-3 h sequentially, and the viewers 2 watch the displayedtraining video sequences 3 a-3 h in step S603. Simultaneously, thecamera tracks the eye movement of the viewers 2 in step S605, and thenthe host PC 111 records the position of each training frame where theviewers 2 gaze in step S607.

Afterward, in step S609, for each training frame, the detected fixationpoints from the viewers 2 are collected to generate a fixation map. Instep S611, the fixation density generator 15 transforms each fixationmap into a fixation density map. In step S613, the feature extractionunit 13 generates four training feature maps (i.e. the color featuremap, motion feature map, orientation feature map, and face feature map)for each training frame of the training video sequences 3 a-3 h based onthe four corresponding feature information. In order to reducecomputation, the training sample selection unit 17 selects one sampleframe from each training video sequence and selects plural sample pixelsfrom the selected sample frame for regressor training in step S615.

After obtaining the training samples, the training unit 18 trains theregression model 19 to learn the correlation relationship between thefixation density and the feature information according to mappingrelationship between the fixation density maps and the training featuremaps of the training samples in step S617. And the work in trainingphase is finished. It is noted that the correlation relationship betweenthe fixation density and the feature information not only can be realtime generated by above steps, but also can be pre-built in theregression model 19 to avoid time consuming of preprocess.

After obtaining the correlation relationship between the fixationdensity and the feature information, it is time to enter into the testphase. In step S619, the feature extraction unit 13 receives a testvideo sequence and generates four tested feature maps for each testframe of the test video sequence based on the four corresponding featureinformation in step S621. Finally, in step S623, the regression model 19maps the tested feature maps into a saliency map according to thetrained correlation relationship, so as to predict the region of visualattention of each test frame. In step S625, the processor (not shown)performs relatively detailed image process on the portion of test framewhich corresponds to the saliency region in the saliency map.

According to the foregoing embodiment, the present invention proposes alearning-based visual attention prediction system and method thereof forvideo signals to provide a computation scheme that firstly obtains thecorrelation relationship between the fixation density and the featureinformation in training phase, and then uses the correlationrelationship to train the regression model 19 in test phase. It predictsvisual attention based on machine learning to determine the relationshipbetween visual attention and the features, so as to avoid perceptualmismatch between the estimated salience and the actual human fixation.

Although specific embodiments have been illustrated and described, itwill be appreciated by those skilled in the art that variousmodifications may be made without departing from the scope of thepresent invention, which is intended to be limited solely by theappended claims.

1. A learning-based visual attention prediction system, comprising: afeature extraction unit configured to receive a test video sequencewhich comprises a plurality of test frames and generate at least onetested feature map for each test frame based on at least one featureinformation; and a regression model, having a correlation relationshipbetween fixation density and the feature information, configured to mapthe at least one tested feature map into a saliency map, which indicatesthe fixation strength of the corresponding test frame, according to thecorrelation relationship.
 2. The system of claim 1, further comprising:a training unit configured to train the regression model to learn thecorrelation relationship between the fixation density and the featureinformation.
 3. The system of claim 2, further comprising: a fixationdata collection unit configured to detect a plurality of fixation pointsgazed in each training frame of a plurality of training video sequencesand collect the fixation points to generate a fixation map for eachtraining frame; and a fixation density generator, coupled to thefixation data collection unit, configured to transform each of thefixation maps into a fixation density map which carries a fixationdensity value for every pixel in the corresponding training frame;wherein, the feature extraction unit further receives the training videosequences and generates at least one training feature map for eachtraining frame of the training video sequences based on the at least onefeature information, and the training unit trains the regression modelaccording to the fixation density maps and the training feature map. 4.The system of claim 3, further comprising: a training sample selectionunit, coupled to the training unit, configured to select at least onesample frame among the training frames of each training video sequence,wherein the density of the fixation points of the selected sample frameis most dense.
 5. The system of claim 4, wherein the training sampleselection unit selects a plurality of sample pixels from the sampleframe, wherein the selected sample pixels are the fixation points wherein relatively dense region of the fixation density map of the sampleframe.
 6. The system of claim 5, wherein the at least one trainingfeature map carries a training feature value of each pixel in thecorresponding training frame, and the training unit trains theregression model according to mapping relationship between the fixationdensity values of the sample pixels and the training feature values ofthe sample pixels.
 7. The system of claim 3, wherein the correlationrelationship is obtained by experimenting on a plurality of viewers, andthe fixation data collection unit comprises: a displayer configured todisplay the training video sequences; a supporting rest configured tosupport the viewers to watch the training video sequences displayed bythe displayer; a camera, fronting the viewers, configured to track theeye movement of the viewers; and a host PC, coupled with the displayerand the camera, configured to control the displayer to display thetraining video sequences and record the positions where the viewers gazein the training frames; wherein, the positions where the viewers gaze inthe training frames are the fixation points.
 8. The system of claim 2,wherein the training unit adopts the support vector regression (SVR)algorithm to train the regression model.
 9. The system of claim 1,wherein the feature information comprises color, motion, orientation, orface.
 10. The system of claim 1, wherein the saliency map has a saliencyregion with relatively large fixation strength.
 11. A learning-basedvisual attention prediction method, comprising: learning a correlationrelationship between the fixation density and at least one featureinformation by training; receiving a test video sequence which comprisesa plurality of test frames; generating at least one tested feature mapfor each test frame based on the at least one feature information; andmapping the tested feature map into a saliency map, which indicates thefixation strength of the corresponding test frame, according to thecorrelation relationship.
 12. The method of claim 11, the step oflearning the correlation relationship comprises: detecting a pluralityof fixation points gazed in each training frame of a plurality oftraining video sequences; collecting the fixation points to generate afixation map for each training frame; transforming each of the fixationmaps into a fixation density map; generating at least one trainingfeature map for each training frame of the training video sequencesbased on the at least one feature information; and learning thecorrelation relationship according to the fixation density maps and thetraining feature map.
 13. The method of claim 12, wherein the step oflearning the correlation relationship further comprises: selecting atleast one sample frame among the training frames of each training videosequence, wherein the density of the fixation points of the selectedsample frame is most dense; and selecting a plurality of sample pixelsfrom the sample frame, wherein the selected sample pixels are thefixation points where in relatively dense region of the fixation densitymap of the sample frame.
 14. The method of claim 13, wherein thefixation density map carries a fixation density value for every pixel inthe corresponding training frame and the training feature map carries atraining feature value of each pixel in the corresponding trainingframe, and the step of learning the correlation relationship furthercomprises; learning the correlation relationship according to mappingrelationship between the fixation density values of the sample pixelsand the training feature values of the sample pixels.
 15. The method ofclaim 12, wherein the correlation relationship is obtained byexperimenting on a plurality of viewers, and the step of detecting thefixation points further comprises: displaying the training videosequences; supporting the viewers to watch the displayed training videosequences; tracking the eye movement of the viewers; and recording thepositions where the viewers gaze in the training frames; wherein, thepositions where the viewers gaze in the training frames are the fixationpoints.
 16. The method of claim 11, adopting the support vectorregression (SVR) algorithm, to train and learn the correlationrelationship.
 17. The method of claim 11, wherein the featureinformation comprises color, motion, orientation, or face.
 18. Themethod of claim 11, wherein the saliency map has a saliency region withrelatively large fixation strength.
 19. The method of claim 18, furthercomprising: performing relatively detailed image process on the portionof the test frame which corresponds to the saliency region.